<html>
<head><meta charset="utf-8"><title>SIMD-enabled utf-8 validation · project-portable-simd · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/index.html">project-portable-simd</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html">SIMD-enabled utf-8 validation</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="213169616"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213169616" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Yoshua Wuyts <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213169616">(Oct 13 2020 at 15:09)</a>:</h4>
<p>Figured folks in the WG might be interested, but a new paper on SIMD-enabled utf-8 validation was just posted on ArXiV <a href="https://arxiv.org/abs/2010.03090">https://arxiv.org/abs/2010.03090</a> -- this got linked to by the author in <a href="https://github.com/rust-lang/rust/issues/68455#issuecomment-706225279">https://github.com/rust-lang/rust/issues/68455#issuecomment-706225279</a></p>
<p>Perhaps one day this could be adapted to be written using portable SIMD instructions ^^</p>



<a name="213184792"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213184792" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213184792">(Oct 13 2020 at 16:51)</a>:</h4>
<p>I (poorly) implemented one of daniel lemire's papers in the past, with just SSSE3 (what's statically available on my macos). Wasn't able to get that great performance but didn't have any special casing for ASCII either... a tricky thing for libcore functionality is that feature detection code needs to be duplicated from stdarch. (it's not core::is_x86_feature_detected, although it <a href="https://docs.rs/core_detect/0.1.0/core_detect/">could be</a>)</p>
<p>maybe this group will solve that idk. would be nice but I think we're putting off most concerns on feature detection.</p>



<a name="213224184"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213224184" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ashley Mannix <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213224184">(Oct 13 2020 at 22:21)</a>:</h4>
<p><span class="user-mention" data-user-id="209168">@Thom Chiovoloni</span> I thought the blocker there was was that feature detection is in <code>std</code> but we need this in <code>core</code>?</p>



<a name="213224252"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213224252" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213224252">(Oct 13 2020 at 22:22)</a>:</h4>
<p>Err, yes. That's what I was trying to say, sorry if it's unclear.</p>



<a name="213224778"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213224778" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ashley Mannix <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213224778">(Oct 13 2020 at 22:28)</a>:</h4>
<p>All good! I've just had _unify <code>std</code>_ in my head lately so am looking for things that are related <span aria-label="smile" class="emoji emoji-1f642" role="img" title="smile">:smile:</span></p>



<a name="213224831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213224831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Caleb Zulawski <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213224831">(Oct 13 2020 at 22:29)</a>:</h4>
<p>Without digging into core_detect, don't you need to know whether your OS supports AVX (for saving the ymm registers at context switches)?</p>



<a name="213226377"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213226377" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213226377">(Oct 13 2020 at 22:41)</a>:</h4>
<p>It's literally the code from std_detect. std_detect requires libstd only on non-x86 targets.</p>



<a name="213226412"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213226412" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213226412">(Oct 13 2020 at 22:41)</a>:</h4>
<p>Which is to say, I assume that's the same as the CPUID bits for AVX, in which case yes.</p>



<a name="213227600"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213227600" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213227600">(Oct 13 2020 at 22:57)</a>:</h4>
<p><span class="user-mention" data-user-id="204346">@Ashley Mannix</span> well, that's certainly a problem that gets easier if "unify std" happens. Even if it isn't stablely exposed via core (because of only 1 platform supporting it) it could still allow libcore functions to use feature detection to provide accelerated impls in some cases</p>



<a name="213227653"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213227653" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Caleb Zulawski <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213227653">(Oct 13 2020 at 22:57)</a>:</h4>
<p>Ah, on x86-64 there's a register that indicates if the OS supports AVX, so no syscall necessary</p>



<a name="213227747"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213227747" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213227747">(Oct 13 2020 at 22:58)</a>:</h4>
<p>doesnt the OS also control (in some manner) what CPUID gives us?</p>



<a name="213227767"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213227767" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213227767">(Oct 13 2020 at 22:58)</a>:</h4>
<p>I don't check that register, but neither does std::is_x86_feature_...</p>



<a name="213227806"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213227806" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213227806">(Oct 13 2020 at 22:59)</a>:</h4>
<p>I was told that CPUID would return the right thing for OSes that disable various SIMD features tho</p>



<a name="213227832"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213227832" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Caleb Zulawski <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213227832">(Oct 13 2020 at 22:59)</a>:</h4>
<p>It does check it, that's what <code>xgetbv</code> does</p>



<a name="213227935"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213227935" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Caleb Zulawski <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213227935">(Oct 13 2020 at 23:00)</a>:</h4>
<p>And cpuid indicates support of that instruction</p>



<a name="213228003"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213228003" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213228003">(Oct 13 2020 at 23:01)</a>:</h4>
<p>ah you're right. i forgot that this is more involved than just reading some values out of cpuid. I check it though <a href="https://github.com/thomcc/core_detect/blob/0bcd5ef3b10626401f61ec16c25f930a4ab76c32/src/os/x86.rs#L203">https://github.com/thomcc/core_detect/blob/0bcd5ef3b10626401f61ec16c25f930a4ab76c32/src/os/x86.rs#L203</a></p>



<a name="213228041"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213228041" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213228041">(Oct 13 2020 at 23:01)</a>:</h4>
<p>and by "I" i mean "the code I copied from std_detect" does</p>



<a name="213312507"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257879-project-portable-simd/topic/SIMD-enabled%20utf-8%20validation/near/213312507" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Henri Sivonen <a href="https://rust-lang.github.io/zulip_archive/stream/257879-project-portable-simd/topic/SIMD-enabled.20utf-8.20validation.html#213312507">(Oct 14 2020 at 16:04)</a>:</h4>
<p><code>encoding_rs</code> has SIMD-accelerated ASCII handling for UTF-8 validation such that it's faster than the standard library for x86_64 and aarch64. It's not as fancy as Daniel Lemire's solution. My intent has been to upstream it to the standard library once the standard library gains portable SIMD.</p>
<p>Unfortunately, ARMv7 being what it is, SIMD wasn't a win there.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>